1. Field of the Invention
The present invention relates in general to an electronic translator system and methods that are particularly suited for facilitating two-way conversations or communications between speaking and/or hearing disabled individuals, and a person capable of normal communications.
2. Description of the Prior Art
In the United States, more than 28 million people, or about 10 percent of the population, have hearing impairments. These individuals often go through life without the opportunity for education, employment or the ability to lead normal lives. In the past 25 years there have been very few technological advancements related to the improvement of education or the communication skills of the deaf, hearing-impaired, blind deaf or hearing mute. Although various types of hearing aids including Cochlear implants and external FM devices have brought some degree of relief to the more profound hearing-impaired, these devices suffer from a number of shortcomings that impede their ability to assist the hearing-impaired individual in improving their communication skills.
For example, cochlear implants are intended to act as artificial cochleae by simulating the sensory hairs that a deaf person is lacking. In response to sensed sound that may come from a person""s voice, the cochlear implant generates electrical impulses that stimulate the hearing receptors in the deaf person""s brain. However, the devices respond not only to a person""s voice, but also to all types of ambient background noise, including other people""s voices, for example. As a result, the cochlear implants generate signals from numerous aural sources that the hearing impaired individual cannot distinguish from each other. A typical outcome is that a hearing impaired individual with a cochlear implant will simply turn the device off since they cannot associate the inputs generated by the implants with words spoken by a person with whom they are trying to communicate.
Other electronic devices have been developed recently that are designed to assist hearing impaired individuals in learning sign language, for example. These types of devices typically receive as input, one or more phrase or words that are entered with a keyboard, for example, and generate animations of signing hand movements that are associated with the entered words and phrase. These types of devices are certainly useful learning tools for the hearing impaired individual, but are of little value for assisting real-time communications. To date, known real-time communication aids for the hearing impaired have been limited either to the aforementioned cochlear implants and other electronic hearing aids, and devices that can convert spoken words to printed text. Speech-to-text translators, while useful for communicating with a hearing impaired individual, cannot be used to teach the hearing impaired individual the sound of the translated words as the text is generated. Until now, a need has therefore remained for a communications and learning aid for hearing impaired and other individuals that does not suffer from the shortcomings of known devices.
The present invention addresses the shortcomings of known communication and learning assistance devices for hearing impaired individuals, or the like, through provision of an electronic translator system and methods that translate input speech and text into multiple streams of data that are simultaneously delivered to a user. Preferably, the data is delivered in audible, visual and, in the case of speech input, text formats. These multiple data streams are delivered to the hearing-impaired individual in a synchronized fashion, thereby creating a cognitive response. In this manner, the present invention allows a deaf or hearing-impaired person to comprehend spoken language, to achieve two-way communications without the requirement of a human translator, to learn natural speech, to learn sign language and to learn to read. The invention is also capable of interconnection with cochlear implants and hearing aids, and alleviates the data overload caused by multiple sounds with varying volume levels being introduced to the human brain simultaneously.
To achieve the foregoing functionality, the system of the present invention, in its most preferred form, is implemented using a personal computer and various conventional peripheral devices. These include, for example, a video display monitor, keyboard, mouse, one or more audio input devices, e.g., microphone, telephone transmitter, etc. and one or more audio output devices, e.g., audio speakers, telephone receiver and hearing enhancement devices used by the deaf or hearing-impaired, such as cochlear implants and hearing aids. The computer is loaded with a number of software programs or modules that perform various translations on received speech or text. In particular, speech received by the audio input device is preferably converted to text, sign language, computer-generated speech for input to the audio devices, and animation or still images that are associated with the words (e.g., an image of an object). The sign language translation is preferably implemented by using the medium of digital movie images in which videos of a person signing words, phrase and finger spelled words are selectively accessed from a database and displayed on the video display monitor. A database of animations and images is also provided, with each of the animations or images being associated with a word or phrase in the text stream. Other databases that are also preferably provided include a phonetic spelling database to enable the phonetic spelling of words to be displayed, and an audio sound database that contains brief audio segments of environmental sounds that are associated with the words or phrase in the text stream.
The computer is preferably selected to have a fast enough processor, e.g., 500 MHz or higher, that the text, signing, audio, animation and other streams can be generated virtually simultaneously in real time, and synchronized with one another. Once synchronized, the data streams are presented to the user concurrently in a method that allows the process of mental comprehension to occur. More particularly, as the spoken words are translated into text, or the words are initially entered as text, they are simultaneously displayed on the video display monitor along with the corresponding signing videos and animation or images, and are supplied to the audio output devices. Preferably, the text words are highlighted on the screen as they are signed and spoken through the audio devices. In this manner, the user of the translator can readily associate sounds generated by a cochlear implant or a hearing aid with the signed and printed words, thus greatly improving the learning and comprehension process. In addition, the problem associated with background noise interference with the cochlear implant or hearing aid is eliminated since the generated sounds correspond only to the words of text.
Preferably, the hearing-impaired person is also able to use the system""s keyboard or mouse to freely converse or respond. Words and phrases typed into the system are converted to computer-generated voice and delivered to the intended audience by the audio speakers or other means, such as the standard or cellular telephone. At this point the system has facilitated two-way conversation between a person capable of normal communications and a hearing-impaired, deaf or mute individual without the requirement of a human translator.
In addition to facilitating two-way conversation, the present invention also enables the deaf and mute to learn to speak. If an individual is physically capable of vocalizing, the invention can be used as a tool to teach linguistics. The user can utilize the computer""s keyboard or mouse to generate words or phrases. The user can then practice repetition of those words or phrases while at the same time hearing a computer generated simulation of their own voice, thereby allowing them the ability to practice and improve speech. Once the user becomes proficient at basic speech, they can use the system""s microphone capability in place of the keyboard or mouse. When the system recognizes the user""s speech patterns it will present them in a synchronized fashion in text, sign language and by computer-generated voice. The generation of these three data streams will support the user""s comprehension of the clarity in which he is speaking thereby affording the opportunity for improvement.
The present invention can also be employed to teach the deaf hearing-impaired, and even individuals with normal hearing and speech skills, to comprehend, communicate and translate multiple variations of sign language. For example, these include but are not limited to American Sign Language, Signed Exact English (SEE), pidgin and various forms of international sign language. As is well known, sign language is a system whereby the hands and arms are used to motion or gesture a communication. For many of the deaf and hearing-impaired, it is the first language learned and crucial to their ability to communicate and achieve an education. The present invention allows sign language to be taught in conjunction with human intervention as in the traditional teacher student scenario, or in a self teach mode whereby the subject can either speak or use the system""s keyboard or mouse to achieve repetitious comprehension of motions and gestures required to create a word or sentence.
Yet another preferred feature of the invention is enabling the deaf hearing impaired or hearing mute, to make and receive standard or cellular telephone calls. To accomplish this task, the system functions as previously described, but allows the option of dialing telephone numbers from the keyboard or with use of the mouse. Once the computer has dialed the number, a digital signal is transmitted through the system that announces the fact that the call is being generated by a hearing or speech impaired individual. The called party will respond either by voice or telephone keypad to acknowledge receipt of the call. The system then converts the voice or digital response to computer generated voice, text and sign language. In this way the deaf hearing impaired or hearing mute individual is able to freely converse by use of the system and its keyboard or mouse. This feature is critical in that it allows the impaired individual to communicate with emergency 911 services. The optional use of a standard telephone coupling device is also provided.